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Abstract 

Probability forecasts are intended to account for the uncertainties in¬ 
herent in forecasting. It is suggested that from an end-user’s point of 
view probability is not necessarily sufficient to reflect uncertainties that 
are not simply the result of complexity or randomness, for example, proba¬ 
bility forecasts may not adequately account for uncertainties due to model 
error. It is suggested that an alternative forecast product is to issue non- 
probabilistic odds forecasts, which may be as useful to end-users, and give 
a less distorted account of the uncertainties of a forecast. Our analysis of 
odds forecasts derives from game theory using the principle that if fore¬ 
casters truly believe their forecasts, then they should take bets at the odds 
they offer and not expect to be bankrupted. Despite this game theoretic 
approach, it is not a market or economic evaluation; it is intended to be 
a scientific evaluation. Illustrative examples are given of the calculation 
of odds forecasts and their application to investment, loss mitigation and 
ensemble weather forecasting. 


1 Introduction 

Our concern here is the quantification of uncertainty in forecasting. Suppose 
our task is to forecast the possibility of overnight temperatures at a particular 
location falling below freezing. This is a task of significant economic impor¬ 
tance, for example, for the salting and gritting of roads, and for preparations to 
mitigate of frost damage in horticulture. Forecasts of this type use atmospheric 
observations and computer models of the physical processes of the atmosphere. 
Uncertainty arises in these forecasts from the complexity of the weather, from 
the sparsity of observations, from the simplifications and inadequacies of the 
computer models, and so on. The suggestion we make here is that in some 
situations, such as using forecasts to mitigate losses, probability may not be 
the best means to quantify the uncertainty of the forecast. We suggest that 
non-probabilistic odds may provide a useful alternative to some forecast users. 

A probability forecast quantifies uncertainty by assigning a probability to 
the occurrence of an event. If the process that determines the outcome of the 
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event is intrinsically random, then there can be a correct or optimal assignment 
to the probability. The forecaster, however, may be uncertain about the cor¬ 
rect probability value to assign, and this leads to consideration of the influence 
of other uncertainties, which we discuss shortly. A well established means for 
quantifying the uncertai nty of probability fo recasting systems are reliabi l ity or 


skill scores. Brieil IQSOl, Mur phy, Winkler Murphy and Winkler . 1977 . 1987 
nkleii . 


iMurphvl . I 1993 I Winkleij . Il994l| . and many others since, have considered using 
scores to assess the skill of a probability foreca st. Skill scores are closely related 
to issues of calibration of p r obabi l ity fore c asts Foster and VohraL ll998L I Palmer , 
2 OOOI Roulston and Smita . 120021 [s mithl . I 1995 I. An alternative assessm ent of 


skill is the economic value of the forecast Granger and Pesaran , 2000l| . Eco¬ 


nomic value is closely related to the idea of wagers. One can imagine a market 
of forecasters who take bets on the outcomes of events they forecast; the fore¬ 
casters that profit the most, or at least avoid bankruptcy, are considered the 
better forecasters. In a competitive market only those fore casters who know 
the t rue probabilities of events will survive in the long term Shafer and Vovk . 


200lj . In the short term , however, merely lucky forecasters can survive, and even 


excel Johnston^ . 20071. Consequently, for a forecaster to be a top performer 
they are forced to act like a bookmaker or marketeer rather than a scientist. 
Scientists aim to learn the true probabilities of events, whereas boo kmakers and 
marketeers respond to the opportunities of a market Levitd . 20041. 

A probability forecast and a skill score together give a more complete picture 
of the uncertainty of a forecasted event. We will argue here, however, that 
probability forecasts can give end-users a misleading picture of uncertainty. We 
suggest a method of avoiding this problem is to issue odds forecasts, which arise 
naturally as wagers. In the next subsections we state our distinction between 
odds and probability, and describe briefly how odds can quantify multiple aspects 
of uncertainty. Section [2] introduces a mathematical formalism for the process 
of issuing odds forecasts, including a brief review of the necessary concepts and 
techniques of game theory. Most importantly we show how the computation of 
odds can be framed as a simple optimisation problem. In section [3] we compute 
odds in four basic forecasting situations and compare these computed odds 
with the corresponding probabilities. In section 0] we describe how odds can 
be employed in investment and loss mitigation. Finally, in section [5] we use 
operational ensemble weather forecasts and station data to present a concrete 
example of issuing odds forecasts. 


1.1 Odds and probability 

Given a complete set of m mutually exclusive events, that is, one and only one 
of the events will occur, we define odds to mean an assignment of real numbers 
gi > 0, i = 1,..., m, to the events. If 9^ = !> then the odds are probabilistic 
odds, and the qi are probabilities. Gasinos and bookmakers assign odds so that 
Si=i g* > 1) which has the consequence that, on average, they should profit 
from the bettors. The excess over one of the sum is sometimes termed “juice”, 
“take”, vigour”. 
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We argue that if a forecaster truly believes their forecasts, then they should 
take bets at the odds they offer and not expect to be bankrupted. This can 
always be achieved by a sufficiently large excess, however, the scientist’s goal 
ought to be to avoid bankruptcy with the smallest excess. 

Taking wagers while avoiding ban kruptcy is a powerful p r inciple. Extending 
early work of Ville it has been used by Foster and Vovk 19991 . Skouras a nd Dawid 
1999l| . Shafer and VovkI 2001 1. Dawid 2004l| and others. Shafer and VovkI 2001 


provides a beautiful development that derives probability theory itself from 
betting principles. The id eas developed in this p aper share a conceptual and 
structural formulation with lShafer and VovkI 200 1| . but there are significant dif¬ 
ferences. Briefly, the differences arise because in our opinion non-probabilistic 
odds are relevant to the immediate and short-term consequences of uncertainty, 
whereas probabilities are generally more relevant to long-term and asymptotic 
uncertainty. 

Bi d-ask spreads in financia l mark ets can be interpreted as non-probabilistic 
odds Levitt . 2004 Johnston'3 . 2007 1 , but our development of odds differs be¬ 
cause we want our forecaster to behave as a scientist and avoid market pressures 
influencing the odds. 


1.2 Levels of uncertainty 

Uncertainty arises from many sources, not just randomness. To appreciate this 
consider Jacob Bernoulli’s foundational example of drawing balls, with replace¬ 
ment, from a urn containing a number of red and black balls. Suppose the 
fraction 6 of red balls in the urn is known, and the process of extracting a ball is 
sufficiently complex to appear uniformly random, that is, on any selection each 
ball in the urn is equally likely to be drawn. In this situation the uncertainty 
is entirely due to the “random” process of selection, and the uncertainty is ad¬ 
equately described by the probability of drawing a red ball, which is 9. We will 
refer to this situation as having first order uncertainty. 

Knowing 9 allows one to make a probability forecast, that is, the probability 
of the event “drawing a red ball” is 9. This is a simple forecast model. Indeed, 
under our assumptions, this is a perfect model, because the model exactly rep¬ 
resents the process, and there is no better model. In a perfect model there is 
only first order uncertainty. 

If the number of red and black balls in the urn is unknown, then a value 
for the fraction of red balls could be inferred from the observed fraction of red 
balls seen in a number of draws. Using ip to forecast the probability of a red ball 
is an imperfect model, there is both first order uncertainty from the randomness 
of the process being forecast, and second order uncertainty due to the value of 
Ip used. 

It is possible to make a further distinction between a perfect model class 
and an imperfect model class. For the urn a value of 0 < ^ < 1 defines a 
class of forecast models. If the assumption of uniform random selection of balls 
is valid, then this is a perfect model class, because the value ip — 9 provides 
a perfect model. If the random selection assumption does not hold, because 
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perhaps the balls are not thoroughly stirred before each selection, then no value 
of Ip provides a perfect model; a perfect model would have to take into account 
the conditional randomness, or even non-random effects, of mixing the urn after 
each replacement. Uncertainty about the correct model class is at least third 
order. 


1.3 At odds with probability 

One goal of forecast ers is to issue accountable, or reliable, or well calib rated, 


V_/X XV_/X V'V_/CXkJ UV.^X O XO IJV-' XOO XXv.^ XXXX UCX V.yXXCX L'Xv.^ v^CXXX L'X CX UV.^V-1 ^ 

probability forecasts Foster and Vohra , 19981 Smithl . Il995l IPalmer , 2000|| , that 


is, if it is forecast that an event E will occur with probability p, then the ob¬ 
served fraction of events E out of all events asymptotically approaches p. It is 
vital to recognise that an imperfect model will almost certainly not provide an 
accountable probability forecast. At best forecasts are only accountable asymp¬ 


totically in a perfect model c lass, but even this is a delicate problem 
19851 Foster and Vohral . Il998 |. 


OakesL 


Suppose that for an urn game a forecaster a provides odds for outcomes of 
draws, and takes bets at these odds. If the forecaster knew 9, and the uniform 
random selection assumption held, then probabilistic odds could be issued where 
a one unit bet on a red ball pays 1 — 9 and betting a black ball pays 9. With 
these probabilistic odds the forecaster does not expect their wealth to increase 
or decrease on average, no matter how skillful the bettors. If the forecaster had 
only an estimate tp, then providing probabilistic odds, where a bet on a red ball 
pays 1 — Ip and a bet on a black ball pays ip, would be unwise. Any bettor 
who knew 9 would almost surely bankrupt the forecaster by always betting a 
fraction 9 of their current wealth on the r ed bal l , and betting the fraction 1 — 9 


of their current wealth on the black ball Kelly . Il95fi| . A similar result is true 


when the bettor just has a better estimate of 9. The fact that ip = 9 is the only 
value of Ip that does not lead asymp totically to certain bank ruptcy can be used 
to define the concept of probability Shafer and VoykI . |2001 |. 

The essential problem with using imperfect models to provide probability 
forecasts is the higher order uncertainties, like model error, distort the odds. 
In particular, we will show that model error often results in under estimating 
the chance of events that have low probability, a so-called base-rate effect. We 
argue that by using non-probabilistic odds forecasts, rather than probability 
forecasts, a forecaster can provide a less distorted forecast that takes into ac¬ 
count model error. (Here we only show how to do this for particular second 
order uncertainties.) 

We argue that if a forecaster aims to provide odds that are as close to 
probabilistic as possible, subject to known information and the model class 
used, then the amount by which the odds exceed one is a measure of how 
certain the forecaster is of their model. The excess is a measure of second order 
uncertainty. (Here we only demonstrate how to do this for a perfect model class 
with particular utility functions.) 
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2 Computation of odds forecasts 


The theoretical framework in which we analyse odds foreca s ting i s game the¬ 


ory 


von Neumann and Morgensterii 11944 iShafer and Vovk 

the Client 


200 ij . There are 


three “players” in our game: the Forecaster, the Client, and Nature. The Client 
determines the structure of the game. Imagine that the Client approaches the 
Forecaster and requests odds for a complete set of m mutually exclusive events 
Ei, i = 1,..., m. Which event actually occurs is determined by Nature, who is 
ignorant and indifferent to the Client-Forecaster negotiations. For our purposes 
we may assume that Nature’s selection of an event is random, or of such com¬ 
plexity that the selection is assumed random by the Client and Forecaster. (By 
making this assumption about Nature we avoid issues of third order uncertain¬ 
ties.) 

When the Client approaches the Forecaster, the Client does not specify in¬ 
terest in any particular event within the set of mutually exclusive events. The 
Forecaster is required to supply odds for all events, based on past observations 
of Nature and the Forecaster’s model. The Forecaster is not a bookmaker, they 
are a scientist. The objective of the scientist Forecaster is to provide odds as 
close to probabilistic as possible given the available information and their model 
class, because by doing so they are aiming to obtain the best forecast model. 

It may be tempting to imagine many clients and forecasters competing in 
a market to determine the best forecast model, but this has the propensity for 
forecasters to adopt the strategies of marketeers and bookmakers. To avoid 
forecasters acting this way we will isolate the Forecaster from market informa¬ 
tion: there will be one Client and the Forecaster is not allowed to know how the 
Client bets, only the choices of Nature. If the Forecaster knew the pat tern of 
the C lient’s bets, then this constitutes an additional information stream Kellvl . 
I956|| . and consequently can be used to improve the Forecaster’s performance 
against the Client, or against Nature if the Client is a more skillful forecaster 
than the Forecaster. Using this additional information stream allows marketeer- 
ing and bookmaking, rather than scientific forecasting. Denying the Forecaster 
knowledge of the Client’s bets forces the Forecaster to rely on available observa¬ 
tions and modelling skill alone. Shortly we will see, however, that the Forecaster 
will need to know a little about the Client’s betting. 


2.1 The Game 


We have a three person game. Nature plays indifferently with no aim. The 
Client aims to accumulate winnings. The Forecaster aims to set as close to 
probabilistic odds as possible, without the Client bankrupting the Forecaster. 
If there were just two events, E and its complement U', then the game can be 
represented by a game matrix Q. 


nature 



E E' 

E 

P -1 

client 

-1 P' 
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The game matrix represents the pay-out to the Client for a one unit bet on an 
event given Nature’s outcome. The odds set by the Forecaster in this case are 
P to 1 for event E, and P' to 1 for event E'. 

There are several variants of this game according to the rules that govern 
the Client’s bets; the variants can influence how the Forecaster sets the odds. 

One rule that can be introduced is the Client’s bets are a fixed size and a 
negligible fraction of the Client’s (and Forecaster’s) total wealth, effectively, 
infinitesimal bets. Alternatively, the Client can bet a substantial fraction of 
their wealth. Other rules that might be introduced govern whether the Client is 
forced to make a bet, regardless of the odds, or whether the Client can split bets 
over several events, or whether there is a minimum size of a bet on any event. 

Rules governing the sizes of bets can influence the Client’s utility function of 
wealth, and consequently influence the pattern of bets. With variable sized 
bets the Client can choose to maximise the growth of wealth, a logarithmic 
utility function, which requires distributing bets over several events. Forced 
bets of a fixed (infinitesimal) size essentially implies a linear utility function of 
wealth, and results in betting only on the event that the Client believes gives 
the maximum pay-out. 

Since the Forecaster does not know the Client’s individual bets, the Fore¬ 
caster must at least know the Client’s utility function of wealth, or the rules 
governing how the Client can bet, which effectively force a utility function onto 
the Client. 

The ultimate challenge for the scientist Forecaster is to compete against a 
Client who knows the true probabilities of Nature, and yet do so without being 
bankrupted. 

2.2 Forced bets of fixed (infinitesimal) size; linear utility 

Much of the analysis in this section is text book game theory [von Neumann and Morgensterri . 
1944| . We first consider the zero-sum game between the Client and Nature and 
determine the Client’s optimal strategy for fixed odds. We then derive an opti¬ 
mal odds assignment of the Forecaster for an arbitrary probability model under 
the assumptions of a perfect model class. 

2.2.1 Optimal Client strategy 

For a complete set of m mutually exclusive events Pj, i = 1,..., m, the m x m 
game matrix Q of the zero-sum game between the Client and Nature (represented 
in 2 X 2 case in ([T])) has Qij — (Sij/qi) — 1, where Sij = 1 ii i = j and zero 
otherwise, and qi is the odds for event Ei, or equivalently, the odds on event Ei 
is “Pi to 1”, where qi = l/(Pi -I-1). Assume that Nature plays a strategy where 
event Ei is chosen randomly with probability Tii. Supposing that the Client 
plays a strategy where event Ei is chosen with probability pi, then the average 
pay-out to the Client is 



(2) 
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Here ’’’i = 1> = 1) but 9* — 1> '''^bh equality if and only if 

the odds are probabilistic. 

The minimax strategy for the Client is to select event with probability 
p*, and has average pay-out H*, where 


Pi = 


qi 


E. 


V* = 


Eiqi 


-1. 


By using the minimax strategy the Client is guaranteed an average pay-out of 
at least V* regardless of the TTi. When the odds are probabilistic, then the 
minimax strategy for the Client has pi = qi and V* = 0. It follows that the 
optimal Client strategy is to choose = 1, when i is the index such that TTi/qi 
is maximal, and pi = 0 otherwise. The average pay-out to the Client is then 

maxi{7ri/gi} - 1. 


2.2.2 Optimal Forecaster strategies 


The forecaster’s aim is to make the odds as probabilistic as possible without 
the client’s winnings accumulating without bound. The forecaster has to allow 
for the possibility that the client is a better forecaster, in the worst case, the 
Client knows Nature’s probabilities tt = (tti, ..., tt^). There is no single optimal 
strategy to achieve the goals we have set for the forecaster, so some additional 
guidance is necessary. 

Since the game matrix Q is symmetric, it can be seen from equation [5] that 
the ideal strategy for the forecaster is to set the odds so that Qi/ Ejli 9j = 
in which case the closest to probabilistic odds are the probabilistic odds qt = tt^. 
This is, of course, a meaningless solution, because the tt^ are unknown. Simply 
taking an estimate fr of Nature’s probabilities tt, and using these as odds, qi = iti, 
is unwise, because at least one 'Kij^i > 1, which will be exploited by the more 
informed Client, who can obtain a better estimate of tt^. 

If the forecaster accepts their forecast model is imperfect, then they will ac¬ 
knowledge there are many likely values of tt. However, if the forecaster assumes 
their model class is perfect, then given data D they can assert that under their 
model Pr(7r | U) oc Pr{D \ 7r)Pr(7r), where Pr(7r) represents prior knowledge 
about possible values of tt. From this the forecaster can compute the average 
loss (pay-out to the client) H(7r, q) for given tt and odds q = (qi, ..., qm), 


V (tt, q) = max 


Qi 


- 1 


The expected loss E{V \ D,q), given the data D and fixed odds q, is 

Jg V (tt, q) Pr(Z) I tt) Pr(7r) dw 


E(VlD,q) = 


Is I Pi'(7r) dTT 


(3) 


(4) 


where S is the simplex Ei ^i = 1- possible optimal strategy for the forecaster 


IS 


lin ^ qi subject to J V (tt, q) Pt{D \ tt) Pr(7r) dir = 0. 


(5) 
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The minimisation attempts to ensure the odds q are as probabilistic as possible. 
The constraint E{V \ D,q) = 0 attempts to ensure that whatever the true tt, 
the odds are such that the client’s average pay-out is zero given the assumed 
distribution of tt. Th e construction of equation ( E]) implies that the client’s 
wealth is a martingale Doobl . 1953 . Williamj . ll991 |. that is, if Wn is the wealth 
of the client after n plays of the game, then E{Wn \ D,q) = Wn-i + E{V \ 
D,q) = Wn-i. 

Observe that the constraint E{V | H, g) = 0 is not, in principle, hard to 
compute. Define 


s = 

{tt: Y,i TTi = 1} , 

(6) 

Si{q) = 

[tt & S\i = argmax{7ri/gi}} , 

(7) 

Aiiq) = 

/ TT Pr(Z? 1 tt) Pr(7r) (Itt, 

JSi{q) 

(8) 

C = 

/ Pi{D 1 tt) Pr(7r) dTT. 

Js 

(9) 


The strategy given by problem ([S]) in equivalent to 

min qi subject to ^ 

Observe that even if Pr(Il | tt) Pr(7r) has a complex form, if it can be computed 
by Monte Carlo methods, then all the Ai{q) and C can be computed simul¬ 
taneously. The main difficulty is that the constraints have to recomputed for 
each q. 


2.3 Forced bets of entire wealth; logarithmic utility 


We now consider the situation where the game rules require the client to bet their 
entire wealth. This is a natural mathematical extension of infinitesimal forced 
bets. With probabilistic odds this game leads to the Kelly betting strategy of 
distributing the client’s entire wealth as bets proportional to the probabilities 
of events. This strategy maximises the growth rate of wealth , or eq uivalently, 
a logarithmic utility function of wealth. However, as iKellvl 195fil | explains, 


when the odds are not probabilistic, then the client should not bet their entire 
wealth, only a fraction of it. We will insist that the client is forced to bet their 
entire wealth. The unfairness of this restriction is balanced by the information 
advantage of the client. If the forecaster tries to make the odds as probabilistic 
as possible, then the odds can still interest the client. 

Once again, suppose there is a complete set of mutually exclusive events Ei, 
i = 1,... ,m and a m x m game matrix Q with Gy = (5y /qi) — 1. The client 
plays nature and is required to bet their entire wealth at each play. Suppose 
that after n plays of this game, the client has total wealth Wn, and at each 
play the client distributes their current wealth as bets in the proportion pi on 
event Ei. Assume nature acts as though the events are chosen randomly, with 
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Ei chosen with probability tt^. Then the expected wealth of the client after n 
plays, given an initial wealth Wq, is 

Consequently, the average rate of growth of wealth is 

^ m 

lim - log(W„) = V] TTi \og{pi/qi). 


It is easily shown that the maximum growth rate occurs when pi = iTi. This is 
the optimal Kelly betting strategy for a client that knows nature’s Hence, 
define G(7r, q) = X]™ i log^iri/qi). 

An appropriate forecaster strategy is to set the closest to probabilistic odds 
so that expected rate of growth of wealth of the client is zero, that is, 


m 

min qi subject to 

i=l 


G(7r, q) Pt{D I tt) Pr(7r) dn = 0. 


( 11 ) 


Equation m implies that the logarithm of the client’s wealth is a martingale, 
that is, E{\ogWn \ D,q) = loglT„_i + ^(G \ D,q) = \ogWn-i- 

Remarkably the optimisation of equation m has an explicit form for the 
solution, which means that in some instances closed form solutions are possible. 
Define, using G as in equation ([5]), 

m 

H = — / ^7r,log(7r*)Pr(i:) I 7r)Pr(7r)d7r, (12) 

i=l 

^ f tTj Pr(L) I tt) Pr(7r) d-TT, (13) 

G Js 

m 

a = id-^7 filog(7fj). (14) 

i=l 


The constraint E{G | D, g) = 0 in (fTT|) is equivalent to '^^i^i^og{qi) = H. 
By straight forward application of Lagrange multipliers it can be shown that 
qi = Tfic" solves the required optimisation (HU. The explicit form of the solu¬ 
tion, in terms of constants that are obtained from integrals, means that in some 
instances closed form solutions are possible. A valid interpretation of this opti¬ 
misation result is that the inflation exponent a is the discrepancy between the 
expected entropy (or information) —H and the entropy implied by the expected 
probabilities tt. 


3 Examples of odds forecasting 

To illustrate the computation of odds we consider two situations forecasting 
binary events. The first situation uses a frequency model, which requires only 
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knowledge of the frequency of past events. The aim of the forecaster is to 
provide odds on a future event. (This is the urn game.) The second situation 
uses a Gaussian model, where the available data is a finite collection of scalar 
measurements assumed to be drawn from a Gaussian distribution. The aim of 
the forecaster is to provide odds on a future measurement being below some 
threshold. We will actually consider four situations because we will compute 
the odds for both linear and logarithmic utility. 


3.1 Frequency model with linear ntility 

Consider the situation where there are two events E and its complement E', 
where Nature selects E with probability tt and E' with probability tt' = 1 — tt. 
A frequency model assumes that all realizations of the events are independent, 
so only the frequency of the event provides any information about tt. The 
forecaster is required to assign odds q and q' to events E and E'. 

Suppose the event E has been observed to occur x times in n realizations. 
Under the frequency model Pr(7r | x,n) oc Pr(a: | tt, n) Pr(7r) = C'^7r^(l — 
7 r)"““ Pr(7r). Suppose the forecaster has no prior information on tt, and so 
assumes a uniform prior Pr(7r) = 1, that is, assumes all values of tt are equally 
likely. 

Following equation (HOD the constraint on q and q' to obtain E{V \ x,n,q,q') = 
0 can be expressed in terms of beta and incomplete beta functions. 



x-\-2) + -(3 ,/ (n 

Q g+g' 


x + l,x + 2) = (}{x + l,n 


x + l), (15) 


where /3a;(a, h) = — t)^~^ dt and /3(a, b) = /?i(a, b). Defining s = q + q' 

and q = ps, so q' = {1 — p)s, then solving equation (fTSl) for s, will conveniently 
transform problem m into a one-dimensional problem, 


min s = 

0 < p<l 


jh^(3p{x + l,n-x + 2) + i^i-p(n 
P{x + 1, n — a; + 1) 


a; + l,a: + 2) 


(16) 


This problem is easily solved numerically using Brent’s method or similar [Press et al 


Figure |T] shows computed odds for various observed frequencies for a small 
number of observations in a table, and for progressively larger numbers of ob¬ 
servations in the graph. Figure [2] shows the total odds s = q + q'. A number of 
interesting, but not unexpected, facts can be seen. For a small number of obser¬ 
vations the odds are far from probabilistic, but they become more probabilistic 
as the number of observations increase. Furthermore, the odds deviate most 
from a probability for the event with a low frequency count, which is consistent 
with the base-rate effect. Observe that meaningful odds are given when x = 0 
or n, and that q —> x/n and q' ^ (n — x)/n as n —> oo. Furthermore, we find 
that for a fixed ratio x/n, q + q' ^ 1 + for some constant c as n —>■ oo. 
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3.2 Frequency model with logarithmic utility 

This situation allows an essentially closed form solution for the odds. Using 
equation (fT^ with Pr(a;,n | tt) = C”7 r'^(l — and Pr(7r) = 1, obtains 


q 

ijj 


a; + 1 
n — X -\- 1 
X + 1 


q' = 


n — X + 1 


n + 2 


Hx+i + 


n — a; + 1 
n + 2 


X +1 

Hn—x+l Hxi+2 


where we use the harmonic numbers \- 

Figure [3] provides a table of computed values of the odds for small values 
of n and graphs the odds for larger values. These computed odds should be 
compared with fig.[TJ Figure S] shows how the total odds q+q' varies with number 
of observations, which should be compared with fig. [2J It is observed that the 
total odds have a smaller excess over unity, and vary less with the observed 
fraction. This is not surprising, because a client who aims to maximise their 
rate of growth of wealth is less likely to exploit forecast errors of low probability 
events. Observe also that the odds converge much faster in this situation, at a 
rate of 1/n, as opposed to l/\/n in the infinitesimal bets case. (The author also 
has closed form expressions for the odds in this case of an arbitrary number of 
events, which will discussed elsewhere.) 


3.3 Gaussian model with linear utility 

Now consider a situation where the forecaster is given scalar observations D = 
{xi, ..., Xn}, with statistics 

= -^{xi-fif = -^xi - 

n ^' n ^' 

i i 

Furthermore, the forecaster has reason to model these as observations of a ran¬ 
dom variable X with a Gaussian distribution for some unknown /i 

and cr; although the forecaster may have additional prior belief Pr(/i, a) in 
the values of fi and cr. The client requires an odds forecast for the events 
E = {X < x} and E' = {X > x} for some fixed x. 

Since we are assuming a perfect model class (to avoid third order uncertainty 
issues), then the true probability tt of the event E is 

TT = Pt{X <x)= [ , ^ dt=^( . 

7-00 Vw V ^ 7 

Whereas the optimisation problem ([5|) that obtains the odds is formulated in 
terms of integrals over the probabilities of the events, in this situation we see 
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these probabilities are determined entirely from x, /i and u. Consequently, it is 
more appropriate to reformulate the optimisation ([5]) it terms of integrals over 
/r and a. This requires reformulating equations m and 

When the forecaster assigns odds q and q' to E and E' respectively, then 
in the worst case where the client knows /i and a (and hence tt) the optimal 
strategy of the client is to bet on 


E if 


E' if 


q + q' 


< $ 


X — fl 


q 


< 1 - $ 


X — II 


q + q' 

and the average pay-out to client is then 

TA/ /I ^ ^ 

V(q,q I X,/X, (t) = max < -O ' 


1 - $ 


X — II 


- 1 . 


Given their model the forecaster will assert that Pr(7r | D) oc Pr(_D 
/X, a) Pr(/x, a) where 


n 

i=i V 

V V^TT^ J 

Just as in section [3.II we once again have a situation involving binary events 
where there is advantage in defining s = q + q', q = ps, q' = (1 — p)s, so that 
problem ([S]) can be transformed to 


min s 

0<p<l 


J_ / , B{p,x) 

C\l-p ^ p 


where 


A{p, x) 

B{p,x) 

C 

M{p,x,a) 

Fill, a) 


OO /*00 


$ 


0 J M{p,x,<t) 
OO pM{p,x,a') 


X — 


0 J —OO 
OO /*oo 


1 - $ 
F(/i, cr) dp da, 


F(/i, a) dp da, 

X — p 


Eip, a) dp da, 


0 —OO 

-I/' 


X — ip), 

Pr(D I p,a)PTip,a). 


The odds can be calculated once the prior Pr(/x, a) has been assigned, how¬ 
ever, the assignment of the prior requires a little care. It is not unreasonable 
to assume no knowledge of p, and hence place a uniform prior on p. On the 
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other hand, one cannot assume a uniform prior for cr, because the integrals will 
diverge. Taking a uniform prior for cr on (0, R), then one finds that s —>■ 1 and 
p —>■ 1/2 as i? —>■ 00 . Essentially a model that freely allows arbitrarily large vari¬ 
ances cannot provide useful forecasts. Consequently, one is required to specify 
a prior for a with a tail that thins sufficiently rapidly. 

Figure [5] shows numerically computed odds for a situation where /t = 0 and 
d = 1. The prior used was Pr(/r,cr) = cre“'^ which is a X"distribution of 
two degrees of freedom. This prior was used because we want d = 1 to be a 
typical observed value. The computed odds were similar for half-normal, F- 
distributions and other similar distributions where d = 1 is a typical observed 
value. By a shift and scale the computed odds shown in fig. [S] provide a general 
solution for arbitrary jl and d. Let Xi = fl+azi so that the Zi have mean zero and 
variance one. These Zi should be modelled as observations of a random variable 
Z = {X — The events of interested are now expressed as E = {Z < zj 

and E' = {Z > z}, where x = fl + az. Hence, if q{z) and s{z) represent the 
odds and total odds for the generic case shown in fig. [51 then the odds in the 
general case are obtained using z = {x — ji)/d. 

Figure [5] shows that the odds on E exceed one for z larger than about 1. Our 
interpretation of this is that given the information available to the forecaster, 
the event E is so likely that offering odds favourable to the client would be loss 
making to the forecaster. The client is therefore offered odds so that to bet 
on E makes a consistent small loss with an rare loss of —1, and to bet on E' 
makes a consistent loss of — 1 and a rare win. 

3.4 Gaussian model with logarithmic utility 

Computation of odds in this situation follow in a similar fashion to the previous 
section, in that the integrals (1121) are reformulated in terms of integrals over p 
and cr. Thus, using the same notation as the previous sections, the odds are 
q = 7fe“ and g' = (1 — 7f)e“ where 

1 nOO nOO 

-I eOO pOO 

2 POO pOO 

Tt = — J J $((z-p)/cr)F(p,cr)dpdcr, 

a = H — TT log(7f) — (1 — 7f) log(l — tt) 

FigureElshows the computed odds for a situation where /} — 0 and d = 1, and 
the prior Pr(p, tr) = (Te~°' These odds should be compared with the linear 
utility situation shown in fig. |5J In comparison it is seen that the logarithmic 
utility function gives odds with smaller excess s and less weight attached to the 
large z values, this is similar to the frequency model, for the same reasons. 
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4 Investment and loss mitigation 

The client who has featured thus far in our analysis is more accurately described 
as a speculative client^ whose primary goal is to profit from inadequacies of the 
forecaster’s predictions. We now introduce the invested client, whose wealth is 
invested in some venture whose profit, costs, and losses are determined in part 
by the outcomes of the events. Think here of the road-gritter, or horticulturist 
who is concerned with the possibility of freezing temperatures. The invested 
client has no desire or ability to challenge the forecaster’s skill, rather they 
wish to use the forecasts to mitigate their losses. The invested client can do 
this by betting against the forecaster; a bet is essentially an insurance policy. 
In this section we analyse how an invested client should bet and show that 
such bets are benehcial to the invested client despite the forecaster’s odds being 
non-probabilistic. Furthermore, we will see that the client deals only with the 
forecaster’s odds, they do not try to normalise the odds to obtain probability 
“estimates”; the odds contain all the information the invested client needs. 

4.1 Simple investment 

Consider a situation where the return on a client’s investment is influenced by 
whether an event E, or its complement E', occurs. Let R be the return on the 
investment when the event E occurs and R' the return on E'. Let tt and tt' , 
where tt + tt' = 1, be the probabilities of the events E and E' respectively, under 
assumption that Nature selects the events at random. The expected return on 
the investment is Rn + R'tt' . 

Now suppose the forecaster provides odds q and q' for the events E and E'. 
The game matrix for betting of these events is o, in terms of the pay-outs 
P = (1/g) — 1 and P' = (1/q') — 1. If the client places bets with the forecaster 
of A > 0 on the event E and A' > 0 on the complement event E', then the 
expected return to the client is 

{R + XP- X')tt + {R' -X + x'p'y. 

If the client chooses the bets A and A' so that R + XP — X' = R' — X + X'P', then 
the client’s return is fixed and independent of tt and tt', indeed independent of 
which event occurs. Under this condition 

i? - i?' = (P' + l)A' - (p + l)A = ^ 

and so, if i? > R', then the client should bet A = 0 and A' = q'{R — R') for a 
return of (l — q')R-\-q'R', and if i? < R', then the client should bet A = q{R' — R) 
and A' = 0 for a return of qR -I- (1 — q)R' ■ 

Since q + q' > 1 the returns with bets are less than the expected return 
without bets, the advantage to the client is that by placing bets the return 
is guaranteed and the risk is transferred to the forecaster. In a market of 
forecasters those that offer the odds closer to being probabilities will attract 
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more clients, but if their odds have not been calculated along the lines we have 
described, then they will be almost surely bankrupted. 


4.2 Mitigating losses 

Consider a situation where a client incurs a loss L if an event E occurs, but 
they can take an action at cost C which if taken results in inclusive mitigated 
losses M. This situation is represented by the following game matrix. 


client 


nature 



E 

E' 

no action 

-L 

0 

action 

-M 

-C 


In this situation it is usual that 0 < C < M < L, although it can happen that 
mitigating the losses includes a reward that more than covers costs of the action 
so that M < C. In either case the optimal strategy for the client is to take the 
action when tt > C/{L + C — M). 

As before a forecaster provides odds on the events E and E'. Suppose the 
client places bets A and A' on E and E' respectively when no action is taken, 
and places bets /i and /i' when the action is taken. The game matrix is the 
following. 



E 

E' 

no action 

-L + XP- X' 

X'P' - X 

action 

-M + /rP - /r' 

-C + ^l'P' -n 


Following the analysis of the previous subsection there are three possibilities: 

(a) Take no action and place bets \ = qL and A' = 0, which results in a fixed 
loss —qL. 

(b) Given C < M, take the action and place bets = q{M — C) and = 0, 
which results in a fixed loss —qM — (1 — q)C. 

(c) Given M < C, take the action and place bets ^ = 0 and fi' = q'{C — M), 
which results in a fixed loss —{l — q')M — q'C. 

It follows that when 0 < C < M < L the client always bets on the event 
occurring and takes the action when q > C/{L + C — M). In the situation 
where 0 < M < C < L the client bets on the event occurring when taking no 
action, and bets on the event not occurring when taking the action, and takes 
the action when qL — {C — M)q' > M. Once again \i q + q' > 1, then the losses 
with bets are more than the expected losses without bets, but the losses are 
fixed and the forecaster takes the risk. 

The important point to note about these results, and those of the previous 
subsection, is the client uses the odds q and q' directly, and does not normalise 
these to obtain probability “estimates” q/{q + q') and q'/{q + q') of the events. 
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All the useful information to the client in contained in the odds, and whether 
the clients refers to both q and q', or just one of these values, depends on their 
circumstances. For example, in the 0 < C < M < L situation the decision to 
take the action is based on q alone, where as, in the 0 < M < C < L situation 
it depends on both q and q', but not in a way that implies normalising them to 
obtain probability “estimates”. 


5 Ensemble Forecasting 

Finally we consider an example of issuing of odds forecasts on temperature 
variations using numerical ensemble weather predictions. A common event of 
interest is whether the temperature at a locality will fall below freezing. These 
events are fairly rare, so we consider a related and more general event of whether 
the temperature at a location at some set lead-time falls more than a certain 
amount below the current temperature at that station. These calculations are 
intended to provide an illustration of odds forecasting using the results we have 
obtained so far. We do not claim that the odds forecast we compute are the 
best or most appropriate, indeed the results suggest otherwise. Odds based on 
kernel density estimates, kernel dressing and the like could provide better odds 
forecasts. 


5.1 Data preparation 

We use London Heathrow (Station 03772) as the locality and Nationa l Center 
for Environmental Prediction GFS ensemble predictions Troth . 2008l| for con¬ 
structing odds forecasts. The GFS model provides a temperature at the station 
by interpolation of the temperatures of the global circulation model’s nearest 
grid-points to the location. Our first step is to prepare suitable time series data 
including bias correction and temporal interpolation. 

The GFS 2-metre-temperature deterministic-forecast initialisation states were 
used to calibrate GFS model temperatures with the station readings as follows. 
Firstly, we compute a cubic spline of the time series of deterministic-forecast 
initialisation states. Secondly, we compute using the spline a model tempera¬ 
ture time-series to pair with the station temperature time-series. Finally, we fit 
by least squares the model 


y = po{x) +pi{x) cos(27r/i/24) + p2{x) sin(27r/i/24). 


(17) 


where y is the station temperature, x the splined model temperature, h the hour 
of the day, and each Pj{x) is a cubic polynomial. This is a twelve parameter 
model. We used times series from the calendar year 2005. The residuals of 
this model had mean 6.03 x 10“^^, skewness 0.0736, kurtosis 0.9453 and stan¬ 
dard error of 1.1159 degrees centigrade. Hence, this model provides a good 
transformation from model state temperature to station temperature with an 
expected error that is very nearly Gaussian and standard deviation of around 
1.1159 degrees centigrade. 
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To obtain ensemble forecasts at any lead-time we compute a cubic spline to 
the time series of each ensemble member, then apply our fitted transform (ED- 

In the following these interpolated and transformed time-series will be re¬ 
ferred to as the adjusted control and the adjusted ensemble forecasts. Figure 0 
shows an example of the station data, adjusted control and adjusted ensemble 
time-series. 

5.2 Odds forecasts 

The goal is to provide odds forecasts on whether the minimum daily temperature 
at one to ten days lead-time will be less than 3 degrees centigrade below the 
adjusted control temperature at the time of issuing the forecasts. In order to use 
the results we have already derived we consider four forecasts. The first forecast 
applies the odds from a frequency model, which is based on counting number 
of adjusted ensemble members below the target threshold. The second forecast 
applies the odds from a Gaussian model, which assumes the adjusted ensemble 
members have a Gaussian distribution. The third forecast is a probabilistic 
forecast obtained by assuming the adjusted ensemble members have a Gaussian 
distribution. The forth forecast is intended to act as the ultimate-challenge 
client. It is not really a forecast at all, but rather the predicted probability of 
station temperature being below the threshold based on a Gaussian distribution 
centred on the adjusted control with standard deviation equal to 1.1159, that 
is, error of the fitted residuals. 

Forecasts one to three represent three competing forecasters, one and two 
providing non-probabilistic odds, three providing probabilistic odds. Forecast 
four is used by the ultimate-challenge client. Nature is always taken to be 
the station temperature. Using the methods described in the previous sections 
odds forecasts can be computed. Figure [8] shows the odds computed using 
the time-series data shown in fig. [3 in the case of a linear utility. Figure [9] 
shows the pay-out of the three forecasters when playing against the ultimate- 
challenge client. Since the challenger is using a linear utility they will bet on 
the temperature being below the threshold if the forecaster’s odds are less than 
the challenger’s odds and visa versa. Observe that the challenger loses more 
bets to forecaster 3, who uses probabilistic odds, than to forecasters 1 and 2, 
who use non-probabilistic odds, however, the winning bets on low probability 
events against forecaster 3 are astronomical. On the other hand, forecasters 1 
and 2 appear to perform about the same. 

We now consider the problem of forecasting whether the minimum temper¬ 
ature at London Heathrow in the 24 hour period from 18:00UTG is more than 
3 degrees centigrade below the control temperature when the forecasts are is¬ 
sued. We will test the performance of the odds forecasts by computing the total 
pay-out to the ultimate-challenge client for the first half of 2005 (183 days) for 
lead times up to ten days. Figures [7] and [8] clearly demonstrate that the proba¬ 
bilistic odds we have considered are not competitive with the non-probabilistic 
odds. Gertainly, large pay-outs result from assigning far too small probability 
to low probability events, but what about the performance at other times? To 
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investigate this question we consider a new non-probabilistic odds forecaster 
(3') that uses the probabilistic odds but sets minimum odds of 0.1, which ef¬ 
fectively caps pay-outs at 10. These capped probabilities are a crude form of 
odds. Odds capping could be applied by our forecasters 1 and 2, although in 
the test considered it makes little difference. Table [1] shows the total pay-out to 
the three non-probabilistic odds forecasters 1,2, and 3', on bets taken from the 
ultimate-challenge client. If forecasters 1 and 2 are allowed odds caps of 0.1 is 
these tests, then the cap only applies on around ten occasions and reduces the 
larger pay-outs at lead times 5, 7, and 10 days to the level of their neighbouring 
lead times. 

A number of observations can be made about the results displayed in table [TJ 
Recall that the aim of the forecasters is to have an expected total pay-out of 
zero. It is immediately clear that odds capping alone is not sufficient to make 
forecaster 3' competitive with forecasters 1 and 2 beyond a lead time of one or 
two days. Forecasters 1 and 2 have similar performance although forecaster 2 
is more successful. In all cases it appears that performance decreases with 
increasing lead time. 


6 Summary, discussion and conclusions 

We have confronted forecasters with the challenge that if they offer a probability 
forecast, then they should be prepared to accept bets at the odds these imply. 
Unless a forecaster has a perfect forecast model, then they would be unwise to 
accept this challenge, because any that do so will almost surely be bankrupted 
by more informed bettors. We argue that probability forecasts fail to account 
for higher order uncertainties, such as model error. Our alternative is to offer 
non-probabilistic odds forecasts obtain using an optimisation principle. The 
excess of the odds reveals the forecaster’s uncertainty about the model. We 
have shown how to compute odds forecasts in several situations, and illustrated 
how odds forecasts could be used in investment, loss mitigation, and weather 
forecasting. 

There are gaps in our development and demonstrations, especially in the 
application to ensemble weather forecasting. A gap occurs because we have 
only shown how to compute odds under the assumption of a perfect model 
class; this assumption eliminates third and higher order uncertainties. Figured 
shows that an assumption of a perfect model class is not well supported in 
our application to ensemble weather forecasting, because during lead-times of 
144 to 180 hours the entire forecast ensemble fails to represent what actually 
happened. This implies the assumption that the ensemble is random selection 
form the distribution of possibilities is false. Nonetheless, table [1] shows that 
our odds calculation is sufficiently conservative at these lead times to cope fairly 
well with this level of uncertainty; pay-outs of around 50 units compared to 200 
units for capped probabilities. The goal, however, was that the pay-outs should 
be zero on average. It would be optimistic to suggest this were the case for 
lead times of 5 days or more. Improvement of these odds forecasts is certainly 
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possible. Either by more careful consideration of the selection of the prior, or by 
abandoning the simple frequency and Gaussian models for a more sophisticated 
model that takes into account the conditional aspects of the weather. In analogy 
to the urn game, a more sophisticated model means looking more closely at the 
mixing process. 

We have argued that odds forecasting has uses in investment and loss miti¬ 
gation, we claim also that it can be used for model assessment. The results of 
table [1] suggest that the Gaussian model is quite successful out to lead times of 5 
days. Kernel-density based models may well do better, extending to longer lead 
times, or having average pay-outs closer to zero. Gomparison with an ultimate- 
challenge client as we do provides a diagnostic of the models performance, and 
furthermore, if a model achieves a zero average pay-out, then excess of the odds 
provide indications of the model’s higher order uncertainty. 
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x/n 

9 

q + q' 

0/1 

0.556 

1.411 

1/1 

0.855 

1.411 

0/2 

0.448 

1.347 

1/2 

0.687 

1.375 

2/2 

0.900 

1.347 

0/3 

0.378 

1.302 

1/3 

0.578 

1.336 

2/3 

0.758 

1.336 

3/3 

0.924 

1.302 

0/4 

0.329 

1.268 

1/4 

0.501 

1.303 

2/4 

0.656 

1.313 

3/4 

0.803 

1.303 

4/4 

0.939 

1.268 


observed frequency x/n 

Figure 1: Odds for frequency model with linear utility function. The table 
shows for frequencies x/n of event E, and the values of q and q + q', as com¬ 
puted from (fTHI). The value of q' can be obtained, by symmetry, from the 
frequency (n — x)ln. Values for larger n are plotted in fig. [T] Only marked 
points are realizable values of the frequency. 
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total odds 



Figure 2: The total odds s = q + q' for frequency model with linear utility 
function. Only marked points are realizable values of the frequency. 
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xju 

9 

9 + 9' 

0/1 

0.382 

1.146 

1/1 

0.764 

1.146 

0/2 

0.277 

1.11 

1/2 

0.558 

1.116 

2/2 

0.832 

1.11 

0/3 

0.217 

1.087 

1/3 

0.438 

1.094 

2/3 

0.656 

1.094 

3/3 

0.87 

1.087 

0/4 

0.179 

1.073 

1/4 

0.359 

1.078 

2/4 

0.54 

1.079 

3/4 

0.719 

1.078 

4/4 

0.894 

1.073 


Figure 3: Odds for frequency model with logarithmic utility function. The table 
shows for frequencies xjn of event £1, and the values of q and 9 + as computed 
from the closed form formula. The value of g' can be obtained, by symmetry, 
from the frequency (n — x)jn. Values for larger n are plotted in fig. [TJ Only 
marked points are realizable values of the frequency. 
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Figure 4: The total odds s = q-\-q' for frequency model with logarithmic utility 
function. Only marked points are realizable values of the frequency. 
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Figure 5: Odds q (dashed) and total odds s = q + q' (solid) for Gaussian model 
with linear utility function, here computed for the generic situation fi = 0 and 
(7=1, prior Pr(/i, a) = and events E = {Z < z} and E' = {Z > z}. The 

standard normal cumulative distribution function is also plotted for comparison 
(dotted). 
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Figure 6: Odds q (dashed) and total odds s = e“ (solid) for Gaussian model with 
logarithmic utility function, here computed for the generic situation /t = 0 and 
(7=1, prior Pr(/i, cr) = and events E = {Z < z} and E' = {Z > z}. The 

standard normal cumulative distribution function is also plotted for comparison 
(dotted). 
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Figure 7: Station temperature time series (+), the adjusted control (dashed) 
and adjusted ensemble forecast time-series (solid). The horizontal line (dashed) 
shows the threshold line for which the odds forecasts are required. The forecasts 
shown were launched 00:00UTC day 40 of 2005. 
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Lead time in hours 


Figure 8: Four computed odds forecasts (linear utility) for the data shown in 
figure[7l Frequency odds (dashed), Gaussian odds (solid), and probabilistic odds 
(dash-dotted). Challenger (dashed) refers to the odds used by the ultimate- 
challenge client against whom the forecasters must compete. 
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Figure 9: The pay-outs of the three forecasters when faced with the ultimate- 
challenge client, for linear utility. Frequency odds (dashed), Gaussian odds 
(solid), and probabilistic odds (dash-dotted). Same data as shown is figures [7] 
andIHl The pay-outs are shown on a linear scale for pay-outs less than one, then 
transitions into a logarithm base 10 scale above. 
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Utility 

Linear 

Logarithmic 

Lead 

1. 

2. 

3'. 

1. 

2. 

3'. 

time 

Freq. 

Cans. 

Capped 

Freq. 

Gaus. 

Capped 

(days) 

odds 

odds 

Prob. 

odds 

odds 

Prob. 

1 

7.3 

47.6 

36.2 

-1.19 

-4.03 

4.94 

2 

65.5 

-11.7 

160 

8.97 

-2 

12.5 

3 

117 

14.9 

233 

20.3 

5.21 

21.6 

4 

78.9 

3.75 

175 

16.6 

5.28 

20.8 

5 

70.9 

68.2 

186 

14.1 

4.54 

18.8 

6 

73.7 

7.71 

166 

15.6 

6.29 

20 

7 

80.5 

82.6 

196 

20.5 

13.1 

26.2 

8 

105 

24.3 

238 

21 

9.89 

27.6 

9 

124 

50.1 

283 

28.5 

18 

34.4 

10 

147 

130 

325 

33.5 

22.9 

38.1 


Table 1: Total pay-out of forecasters on bets taken from the ultimate-challenge 
client on the event that the minimum temperature at London Heathrow in the 
24 hours period from 18:00UTC is more than 3 degrees centigrade below the 
control temperature when the forecasts are issued. Total computed over first 
half of 2005. In the case of logarithmic utility the pay-out is expressed as the 
logarithm base 10. 
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